{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Transfer learning with PyTorch \n",
    "\n",
    "In this example we'll implement fine-tuning and feature extracting transfer learning using the CIFAR-10 dataset. \n",
    "\n",
    "_This example is partially based on_ [https://github.com/pytorch/tutorials/blob/master/beginner_source/transfer_learning_tutorial.py](https://github.com/pytorch/tutorials/blob/master/beginner_source/transfer_learning_tutorial.py).<br/>\n",
    "_The licensing information and the author of the base version are:<br/>\n",
    "License: BSD<br/>\n",
    "Author: Sasank Chilamkurthy_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start with the imports:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "import torchvision\n",
    "from torchvision import models, transforms"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll continue with the training and validation batch size:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 50"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we'll define the train data loader. `train_data_transform` resizes the CIFAR images to match the transfer learning model input size. The function also applies basic data augmentation techniques:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Files already downloaded and verified\n"
     ]
    }
   ],
   "source": [
    "train_data_transform = transforms.Compose([\n",
    "    transforms.Resize(224),\n",
    "    transforms.RandomHorizontalFlip(),\n",
    "    transforms.RandomVerticalFlip(),\n",
    "    transforms.ToTensor(),\n",
    "    transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])\n",
    "])\n",
    "\n",
    "train_set = torchvision.datasets.CIFAR10(root='./data',\n",
    "                                         train=True,\n",
    "                                         download=True,\n",
    "                                         transform=train_data_transform)\n",
    "\n",
    "train_loader = torch.utils.data.DataLoader(train_set,\n",
    "                                           batch_size=batch_size,\n",
    "                                           shuffle=True,\n",
    "                                           num_workers=2)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll continue with the validation data loader:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Files already downloaded and verified\n"
     ]
    }
   ],
   "source": [
    "val_data_transform = transforms.Compose([\n",
    "    transforms.Resize(224),\n",
    "    transforms.ToTensor(),\n",
    "    transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])\n",
    "])\n",
    "\n",
    "val_set = torchvision.datasets.CIFAR10(root='./data',\n",
    "                                       train=False,\n",
    "                                       download=True,\n",
    "                                       transform=val_data_transform)\n",
    "\n",
    "val_order = torch.utils.data.DataLoader(val_set,\n",
    "                                        batch_size=batch_size,\n",
    "                                        shuffle=False,\n",
    "                                        num_workers=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we'll select the device (try for GPU, default on CPU):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's define the training procedure:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_model(model, loss_function, optimizer, data_loader):\n",
    "    # set model to training mode\n",
    "    model.train()\n",
    "\n",
    "    current_loss = 0.0\n",
    "    current_acc = 0\n",
    "\n",
    "    # iterate over the training data\n",
    "    for i, (inputs, labels) in enumerate(data_loader):\n",
    "        # send the input/labels to the GPU\n",
    "        inputs = inputs.to(device)\n",
    "        labels = labels.to(device)\n",
    "\n",
    "        # zero the parameter gradients\n",
    "        optimizer.zero_grad()\n",
    "\n",
    "        with torch.set_grad_enabled(True):\n",
    "            # forward\n",
    "            outputs = model(inputs)\n",
    "            _, predictions = torch.max(outputs, 1)\n",
    "            loss = loss_function(outputs, labels)\n",
    "\n",
    "            # backward\n",
    "            loss.backward()\n",
    "            optimizer.step()\n",
    "\n",
    "        # statistics\n",
    "        current_loss += loss.item() * inputs.size(0)\n",
    "        current_acc += torch.sum(predictions == labels.data)\n",
    "\n",
    "    total_loss = current_loss / len(data_loader.dataset)\n",
    "    total_acc = current_acc.double() / len(data_loader.dataset)\n",
    "\n",
    "    print('Train Loss: {:.4f}; Accuracy: {:.4f}'.format(total_loss, total_acc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll continue with the test procedure:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def test_model(model, loss_function, data_loader):\n",
    "    # set model in evaluation mode\n",
    "    model.eval()\n",
    "\n",
    "    current_loss = 0.0\n",
    "    current_acc = 0\n",
    "\n",
    "    # iterate over  the validation data\n",
    "    for i, (inputs, labels) in enumerate(data_loader):\n",
    "        # send the input/labels to the GPU\n",
    "        inputs = inputs.to(device)\n",
    "        labels = labels.to(device)\n",
    "\n",
    "        # forward\n",
    "        with torch.set_grad_enabled(False):\n",
    "            outputs = model(inputs)\n",
    "            _, predictions = torch.max(outputs, 1)\n",
    "            loss = loss_function(outputs, labels)\n",
    "\n",
    "        # statistics\n",
    "        current_loss += loss.item() * inputs.size(0)\n",
    "        current_acc += torch.sum(predictions == labels.data)\n",
    "\n",
    "    total_loss = current_loss / len(data_loader.dataset)\n",
    "    total_acc = current_acc.double() / len(data_loader.dataset)\n",
    "\n",
    "    print('Test Loss: {:.4f}; Accuracy: {:.4f}'.format(total_loss, total_acc))\n",
    "\n",
    "    return total_loss, total_acc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we'll define 2 functions that build transfer learning models for either feature extacting, or fine-tuning. Both models use the `torchvision.models.resnet18` ImageNet pretrained model. We'll start with feature extracting, which \"locks\" all model parameters (weights) except for the final fully-connected layer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def tl_feature_extractor(epochs=5):\n",
    "    # load the pre-trained model\n",
    "    model = torchvision.models.resnet18(pretrained=True)\n",
    "\n",
    "    # exclude existing parameters from backward pass\n",
    "    # for performance\n",
    "    for param in model.parameters():\n",
    "        param.requires_grad = False\n",
    "\n",
    "    # newly constructed layers have requires_grad=True by default\n",
    "    num_features = model.fc.in_features\n",
    "    model.fc = nn.Linear(num_features, 10)\n",
    "\n",
    "    # transfer to GPU (if available)\n",
    "    model = model.to(device)\n",
    "\n",
    "    loss_function = nn.CrossEntropyLoss()\n",
    "\n",
    "    # only parameters of the final layer are being optimized\n",
    "    optimizer = optim.Adam(model.fc.parameters())\n",
    "\n",
    "    # train\n",
    "    test_acc = list()  # collect accuracy for plotting\n",
    "    for epoch in range(epochs):\n",
    "        print('Epoch {}/{}'.format(epoch + 1, epochs))\n",
    "\n",
    "        train_model(model, loss_function, optimizer, train_loader)\n",
    "        _, acc = test_model(model, loss_function, val_order)\n",
    "        test_acc.append(acc)\n",
    "\n",
    "    plot_accuracy(test_acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll continue with the fine-tuning model, which trains all weights of the network:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def tl_fine_tuning(epochs=5):\n",
    "    # load the pre-trained model\n",
    "    model = models.resnet18(pretrained=True)\n",
    "\n",
    "    # replace the last layer\n",
    "    num_features = model.fc.in_features\n",
    "    model.fc = nn.Linear(num_features, 10)\n",
    "\n",
    "    # transfer the model to the GPU\n",
    "    model = model.to(device)\n",
    "\n",
    "    # loss function\n",
    "    loss_function = nn.CrossEntropyLoss()\n",
    "\n",
    "    # We'll optimize all parameters\n",
    "    optimizer = optim.Adam(model.parameters())\n",
    "\n",
    "    # train\n",
    "    test_acc = list()  # collect accuracy for plotting\n",
    "    for epoch in range(epochs):\n",
    "        print('Epoch {}/{}'.format(epoch + 1, epochs))\n",
    "\n",
    "        train_model(model, loss_function, optimizer, train_loader)\n",
    "        _, acc = test_model(model, loss_function, val_order)\n",
    "        test_acc.append(acc)\n",
    "\n",
    "    plot_accuracy(test_acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's define the `plot_accuracy` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_accuracy(accuracy: list):\n",
    "    \"\"\"Plot accuracy\"\"\"\n",
    "    plt.figure()\n",
    "    plt.plot(accuracy)\n",
    "    plt.xticks(\n",
    "        [i for i in range(0, len(accuracy))],\n",
    "        [i + 1 for i in range(0, len(accuracy))])\n",
    "    plt.ylabel('Accuracy')\n",
    "    plt.xlabel('Epoch')\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we can put everything together and run the training. Let's start with the feature extracting model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "Train Loss: 1.0454; Accuracy: 0.6422\n",
      "Test Loss: 0.7246; Accuracy: 0.7565\n",
      "Epoch 2/5\n",
      "Train Loss: 0.8640; Accuracy: 0.7000\n",
      "Test Loss: 0.7097; Accuracy: 0.7575\n",
      "Epoch 3/5\n",
      "Train Loss: 0.8363; Accuracy: 0.7064\n",
      "Test Loss: 0.6952; Accuracy: 0.7622\n",
      "Epoch 4/5\n",
      "Train Loss: 0.8144; Accuracy: 0.7114\n",
      "Test Loss: 0.6752; Accuracy: 0.7651\n",
      "Epoch 5/5\n",
      "Train Loss: 0.8179; Accuracy: 0.7145\n",
      "Test Loss: 0.6859; Accuracy: 0.7624\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEGCAYAAABy53LJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3dd3xV9f3H8dcni7BnCJuwCYKAho36A5ytiqIyalEcxYF7tNpa29r+ah0Vq+LAgVKtLEGwPxfDqggKAcIMG4EwMoAQQghZ398fufiIlJEr9+bcm7yfj0ce5J5778n7XgLve875nvM15xwiIiLlFeF1ABERCS8qDhER8YuKQ0RE/KLiEBERv6g4RETEL1FeB6gIjRo1cgkJCV7HEBEJK8uWLctyzsUdv7xKFEdCQgLJyclexxARCStmtv1Ey7WrSkRE/KLiEBERv6g4RETELyoOERHxi4pDRET8ouIQERG/qDhERMQvKg4RCQs79+fx/pIdFBaXeB2lyqsSJwCKSHhL2ZnNLW8vZd/hAuak7Obl68+hfs0Yr2NVWdriEJGQNnddOiMnLqZGtUgevawzy3Yc4MoJC1m/N8fraFWWikNEQtbkxd9z2z+T6RRfm5l3DOC2C9oxdWxfjhaWMOzlRXy6Zq/XEaskFYeIhJySEseTH6fy+Oy1DO7cmPfH9iWudjUAeraqz0d3D6RDfG1uf3cZz8/bSEmJpsCuSCoOEQkp+YXF3DNlBa99tZXRfVvz2ugkasT8+HBsfJ1Ypo7ty7BzmvP8vE3c+d5yDh8t8ihx1aPiEJGQkZ1XwA1vLuHfq/bw6GWdeWLoWURG2AkfGxsdyd+v685jP0/k83V7ueaVRezcn1fBiasmFYeIhISd+/MY9soiUnZm8+Kontx2QTvMTlwax5gZt57Xlrdv6s3u7CNc+dJCFm3JqqDEVZeKQ0Q8tyotm6tf/oZ9uQW8e2sfrujezK/nn98xjtl3DaRhrWqMfnMJkxd/j3M67hEsKg4R8dT81HRGvPYtsdGRfHBHP3q3afCT1tOmUU1m3dmfQZ3ieHz2Wn47azUFRTpZMBhUHCLimXe/3c6vJifTvnEtZt7Zn/aNa5/R+mrHRjNxdBLjBrXj/SU7+cXr35J56GiA0soxKg4RqXAlJY6/fbKexz5cw6BOjZkyti+Na8cGZN0REcbDl3TmxVE9WbP7IENfWsiaXQcDsm4ppeIQkQp1tKiYe6em8OqXW7i+TyteG30uNasF/upHV3Rvxozb+wNw7auLmLNyd8B/RlWl4hCRCnMwr5DRby7ho5W7+c2lnfnLVV2Jigzef0Ndm9dl9l0D6da8Lve8v4KnPl1PsU4WPGMqDhGpEDv353HNq4tI2ZHNP0b24I7/Of1w20CIq12N927ty6jerXjlP1v41eRkDuUXBv3nVmYqDhEJutVpB7n65UVk5OQz+ZbeDO3RvEJ/fkxUBH+9uit/HnoWX23M5OqXF7Et63CFZqhMVBwiElQL1qcz/LXFVIuKYOad/enbtqEnOcyM0f0S+OctfdiXe5ShLy3kq42ZnmQJdyoOEQmaf323g1vfSaZd49JzLM50uG0g9GvXkDl3DaRZveqMmbSEN77eqpMF/aTiEJGAKylxPP3pen47azUXdIxj6th+NK4TmOG2gdCyQQ0+uKM/l5zVhL/8XyoPTl9JfmGx17HChopDRALqaFEx909L4eX/bGFU71a8fkNSUIbbnqma1aKY8ItzuP/CjsxcvosRE78lPSff61hhQcUhIgFzMK+QG99awuyU3Tx8SSf+enVwh9ueqYgI494LO/DqL89lU/ohrnhxISt2HPA6VsgL3b9REQkraQfyuPbVRSzbfoDnR/Rg3KD2FTLcNhAu7dqEmXf2p1p0BCNe+5YZy9K8jhTSVBwicsbW7Codbrs3J593bu7NVT0rdrhtIHRuUoc54waSlFCfh6av5M//XkdRsS6SeCJBLQ4zu9TMNpjZZjN75AT3jzezFN/XRjPLLnNfKzP73MxSzWydmSUc99wXzCw3mPlF5PS+2JDB8NcWEx1hfHBHf/q3a+R1pJ+sfs0Y3rm5N2P6J/Dmwm3c9PZSDubpZMHjBa04zCwSmABcBnQBRplZl7KPcc7d75zr4ZzrAbwIzCxz92TgGedcItAbyCiz7iSgfrCyi0j5vL+kdLhtm0Y1mTVuAB3jvR9ue6aiIyP445Vn8dQ13fh26z6GTljI5oxDXscKKcHc4ugNbHbObXXOFQBTgKGnePwo4H0AX8FEOefmAjjncp1zeb77IoFngF8HMbuInIJzjmc/28CjM1czsH0jpt7Wj/gQGm4bCCN6tWLK2L7kHi3mqgmLmJ+a7nWkkBHM4mgO7CxzO8237L+YWWugDbDAt6gjkG1mM81shZk94ysMgLuAOc65PUHKLSKnUFBUwgPTVvLSF5sZ2aslb9yYRK0QHG4bCOe2bsCcuwbQplFNbp2czIQvNutkQULn4PhIYIZz7tgZOFHAecBDQC+gLTDGzJoB11G6W+uUzGysmSWbWXJmpi4rIBIIB4+UDredtWIXD13ckSeHdSM6hIfbBkKzetWZfns/ruzejGc+28Dd76/gSEHVPlkwmB8TdgEty9xu4Vt2IiOBcWVupwEpzrmtAGb2IdAX2Au0Bzb7hvnVMLPNzrn2x6/QOTcRmAiQlJSkjwgiZ2hX9hFumrSEbVmHeW54d4ad08LrSBUmNjqS50f0ILFpHZ76dD3bsg4z8YYkmter7nU0TwTzo8JSoIOZtTGzGErLYc7xDzKzzpQe6F583HPrmVmc7/ZgYJ1z7v+cc02ccwnOuQQg70SlISKBtXb3Qa6e8A17svN556beVao0jjEzbr+gHW/emMSOfXlc+eJClmzb73UsTwStOJxzRZQej/gMSAWmOefWmtkTZnZlmYeOBKa4MjsOfbusHgLmm9lqwIDXg5VVRE7uy42ZDH91MZERxow7+tO/ffgOtw2EwZ3jmTVuAHWrR3P9G9/yr+92eB2pwllVONCTlJTkkpOTvY4hEnamLt3Bb2etoWN8bSaN6UWTupVr5NSZOHikkHveX8GXGzMZ3bc1j1/RpdId7zGzZc65pOOXV65XKSIB4Zzjuc838JsPVjOgfSOm3dZXpXGcutWjeWtML247vy3//HY7o9/8jv2HC7yOVSFUHCLyIwVFJTw4fSUvLNjM8KQWvHljErVjo72OFZIiI4xHf5bI+BHdWb4jmytfWkjqnhyvYwWdikNEfpCTX8hNby9h5vJdPHBRR5665uxKt/slGK7u2YLpt/WjsLiEYS8v4pPVlfs0M/1GiAgAu7OPcN0ri/lu636eva479wzpEDZXtw0F3VvW46O7BtK5aW3ueG85z83dSElJ5TyGrOIQEdbtzmHYy4vYnX2Et2/qzbXnVr3htoHQuE4sU8b25bpzW/DC/E3c8d4yco8WeR0r4FQcIlXcVxszGf5a6WlU0+/ox8AOVXu47ZmqFhXJ09eezeOXd2FeagbXvLyIHfvyvI4VUCoOkSpsWvJObn57KS3qV2fWuP50blLH60iVgplx88A2vHNTb/bm5HPlhIUs2pzldayAUXGIVEHOOcbP3civZ6yiX7uGTL+9H03rVs3LZwTTwA6NmD1uAHG1qjH6rSW8/c22SnGRRBWHSBVTWFzCwzNW8Y/5m7j23Ba8NaaXhtsGUUKjmsy8sz+DOjXmjx+t45EPVnO0KLwvkqjiEKlCDuUXcvPbS5mxLI37LuzAM9dquG1FqB0bzcTR53L34PZMTd7JL17/joxD+V7H+sn0GyNSRew5eITrXl3M4i37eObas7nvwo4abluBIiKMBy/uxIRfnMO63TkMfekbVqcd9DrWT6LiEKkCUvfkcPWERaQdOMJbY3pxXVLL0z9JguLnZzdlxh39iDDj2lcXMTvlZLNNhC4Vh0glt3BTFsNfXYzDMe22fpzfMe70T5KgOqtZXebcNYDuLetx75QU/vbJeorD6GRBFYdIJTZjWRpjJi2hef3qzLpzAF2aabhtqGhYqxrv3tKH6/u04tUvt3DrO0vJyS/0Ola5qDhEKiHnHP+Yt4mHpq+kT9sGTLu9H82q6Gx1oSwmKoL/vbobf7mqK19vyuKqCd+wNTPX61inpeIQqWQKi0v4zQerGD9vI8POac6kMb2po+G2Ie2XfVvz3q19yM4rZOiEb/jPhgyvI52SikOkEjk23HZachr3DOnA36/rTkyU/pmHgz5tGzJ73ABa1K/BzW8v5bUvt4TsyYL6jRKpJNJz8hn+2rcs2rKPp685mwcu0nDbcNOyQQ0+uKMfl3VtypOfrOeBaSvJLwy9kwWjvA4gImduw95D3DRpCQePFPLWmF5coJFTYatGTBQv/aInnRfU5u9zN7IlM5eJo5NCagZGbXGIhLlFm7O49pVFFJU4pt3eT6VRCZgZdw/pwMTR57IlI5crXlrI8h0HvI71AxWHSBibuTyNGyctoWm9WGaNG8BZzep6HUkC6OKzmjBr3ABqxEQy8rVvmZ680+tIgIpDJCw553hpwSYemLaSpNYNmH57f5pruG2l1DG+NrPHDaB3mwY8PGMVf/poLUXFJZ5mUnGIhJnC4hIenbmaZz/fyNU9m/POzb2pW13DbSuzejViePumXtw8oA2TvvmeMZOWkp1X4FkeFYdIGMk9WsSt7yQzZelO7h7cnueGa7htVREVGcHjV3Th6WvPZsm2/Qyd8A0b0w95kkW/cSJhIj0nn+GvLmbh5iyeHNaNBy/upOG2VdDwpJa8P7YveQXFXD3hGz5fu7fCM6g4RMLAxvRDDHt5Ed/vO8wbNyYxqncrryOJh85tXZ85dw2gXeNajP3nMl6cv6lCTxZUcYiEuEVbsrjmlUUUFJcw7bZ+DOrU2OtIEgKa1q3OtNv6cVWPZvx97kbu+tcK8gqKKuRn6wRAkRD24YpdPDxjJQkNazLppl60qF/D60gSQmKjIxk/ogddmtXhb5+sZ2vWYV6/4dyg/55oi0MkBDnnmPDFZu6bmsK5resz4/b+Kg05ITNj7PnteGtML9IO5HHlS9/w3dZ9Qf2ZKg6REFNUXMJvZ63hmc82MLRHs9LhtjU03FZO7X86NebDcQOoVyOa69/4jne/3R60n6XiEAkhh48W8avJyby/ZAfjBrVj/PAeVIuK9DqWhIl2cbX4cNwAzuvQiMc+XMPvZq2moCjwJwuqOERCREZOPiMmLuarTVn89epuPHxJZyIiNNxW/FMnNpo3buzF7Re0Y1ryTjbsDfy5Hjo4LhICNqUfYsykpRzIK+CNG5IY1Fkjp+Sni4wwHrmsM7/o3YpWDQN/bEzFIeKxb7fuY+zkZGKiIpk6th/dWuhChRIYwSgNUHGIeGp2yi4enr6KVg1rMGlML1o20MgpCX0qDhGPTFu6k19/sIo+bRowcXSSRk5J2FBxiHhgc0Yuj89Zw8D2jXhzTJJGTklYCeqoKjO71Mw2mNlmM3vkBPePN7MU39dGM8suc18rM/vczFLNbJ2ZJfiWv+db5xoze8vM9DFNwkpBUQn3TV1BjZgonhvRXaUhYSdoxWFmkcAE4DKgCzDKzLqUfYxz7n7nXA/nXA/gRWBmmbsnA8845xKB3kCGb/l7QGegG1AduDVYr0EkGJ6ft5E1u3J4clg3GtcOnXmkRcormFscvYHNzrmtzrkCYAow9BSPHwW8D+ArmCjn3FwA51yucy7P9/3HzgdYArQI4msQCajvtu7jlS+3MLJXSy45q4nXcUR+kmAWR3Og7AS5ab5l/8XMWgNtgAW+RR2BbDObaWYrzOwZ3xZM2edEA6OBT0+yzrFmlmxmyZmZmWf4UkTOXE5+IQ9MW0nrBjX4/eVdTv8EkRAVKmeOjwRmOOeKfbejgPOAh4BeQFtgzHHPeRn4yjn39YlW6Jyb6JxLcs4lxcXFBSe1iB/+MHste3PyGT+iBzWraVyKhK9gFscuoGWZ2y18y05kJL7dVD5pQIpvN1cR8CFwzrE7zewPQBzwQEATiwTJnJW7mbViF/cM7kDPVvW9jiNyRoJZHEuBDmbWxsxiKC2HOcc/yMw6A/WBxcc9t56ZHdtUGAys8z3+VuASYJRzLvBX7xIJsF3ZR/jdrNWc06oe4wa18zqOyBkLWnH4thTuAj4DUoFpzrm1ZvaEmV1Z5qEjgSmuzLyHvl1WDwHzzWw1YMDrvrtfBeKBxb5hvI8H6zWInKmSEseD01IoKXGMH9GDqMhQ2Tss8tMFdUerc+5j4OPjlj1+3O0/nuS5c4GzT7BcO4clbLyxcCvfbt3P09eeTeuGNb2OIxIQ+vgjEiRrdx/kmc82cOlZTbjuXI0al8pDxSESBPmFxdw3JYX6NWJ4clg3zDSvhlQepy0OM7vbzDQMRMQPf/tkPZsycnn2uu7UrxnjdRyRgCrPFkc8sNTMpvmuPaWPTiKn8OXGTN5e9D03DUjg/I46h0gqn9MWh3PuMaAD8CalJ+FtMrO/mpnGFYocZ//hAh6avpKO8bX4zaWdvY4jEhTlOsbhGyq71/dVROl5FzPM7OkgZhMJK845HvlgFQfzCnl+RE9io3XVW6mcTju01czuBW4AsoA3gIedc4VmFgFsAn4d3Igi4WFa8k4+X5fO736WSJdmdbyOIxI05TknogEwzDm3vexC51yJmV0enFgi4eX7rMP86aN19G/XkFsGtvE6jkhQlWdX1SfA/mM3zKyOmfUBcM6lBiuYSLgoKi7hvqkpREUYfx/enYgIjR+Ryq08xfEKkFvmdq5vmYgALy7YTMrObP46rBtN61b3Oo5I0JWnOOy460iVoLnKRQBYtv0AL32xmWE9m3P52c28jiNSIcpTHFvN7B4zi/Z93QtsDXYwkVCXe7SIB6al0LRuLH8aepbXcUQqTHmK43agP6VzaaQBfYCxwQwlEg6e+GgtO/fn8dzwHtSOjfY6jkiFOe0uJ+dcBqWXPhcRn0/X7GFachrjBrWjd5sGXscRqVDlOY8jFrgFOAuIPbbcOXdzEHOJhKz0nHwembmas1vU5b4LO3odR6TClWdX1T+BJpTOuvclpVPAHgpmKJFQVVLieGj6SvILixk/ogfRmphJqqDy/Na3d879HjjsnHsH+DmlxzlEqpx3Fn/P15uyeOznXWgXV8vrOCKeKE9xFPr+zDazrkBdoHHwIomEpo3ph3jyk/UM6dyY6/u08jqOiGfKcz7GRN98HI8Bc4BawO+DmkokxBwtKuae91dQJzaKp649WxMzSZV2yuLwXcgwxzl3APgKaFshqURCzN8/38j6vYd488YkGtWq5nUcEU+dcleV7yxxXf1WqrRFm7N4/eut/LJvK4YkxnsdR8Rz5TnGMc/MHjKzlmbW4NhX0JOJhICDeYU8OH0lbRrV5Hc/6+J1HJGQUJ5jHCN8f44rs8yh3VZSyTnn+N2Hq8k8dJRZdw6geowmZhKB8p05rskFpEr6MGUX/161h4cv6US3FnW9jiMSMspz5vgNJ1runJsc+DgioWHn/jwe/3AtvRLqc/sF7byOIxJSyrOrqleZ72OBIcByQMUhlVJxiePBaStxwHPDexCpiZlEfqQ8u6ruLnvbzOoBU4KWSMRjr365hSXf7+e54d1p2aCG13FEQs5PudDOYUDHPaRSWp12kPFzN3L52U25umdzr+OIhKTyHOP4iNJRVFBaNF2AacEMJeKFIwXF3Dt1BXG1q/G/V3XT2eEiJ1GeYxzPlvm+CNjunEsLUh4Rz/zvx+vYlnWY927pQ90amphJ5GTKUxw7gD3OuXwAM6tuZgnOue+DmkykAi1Yn8673+5g7Plt6d++kddxREJaeY5xTAdKytwu9i0TqRSyco/y6xmr6NykNg9erImZRE6nPMUR5ZwrOHbD931M8CKJVBznHL+ZsYqc/CJeGNWTalE6O1zkdMpTHJlmduWxG2Y2FMgKXiSRivOvJTuYvz6DRy/rTMf42l7HEQkL5TnGcTvwnpm95LudBpzwbHKRcLIlM5c//3sd53VoxI39EryOIxI2ynMC4Bagr5nV8t3ODXoqkSArLC7hvikpVI+O5NnruhOhs8NFyu20u6rM7K9mVs85l+ucyzWz+mb2l4oIJxIs/5i3idW7DvLksG7E14n1Oo5IWCnPMY7LnHPZx274ZgP8WXlWbmaXmtkGM9tsZo+c4P7xZpbi+9poZtll7mtlZp+bWaqZrTOzBN/yNmb2nW+dU81MB+rFL0u/38/L/9nM8KQWXNq1qddxRMJOeYoj0sx+mCvTzKoDp50708wigQnAZZSebT7KzH40E45z7n7nXA/nXA/gRWBmmbsnA8845xKB3kCGb/lTwHjnXHvgAHBLOV6DCAA5+YXcNyWFlg1q8PgVZ3kdRyQslac43gPmm9ktZnYrMBd4pxzP6w1sds5t9Q3hnQIMPcXjRwHvA/gKJso5NxdKj6s45/Ks9BoQg4EZvue8A1xVjiwiAPxxzlr25uQzfkQPalUrz9gQETneaYvDOfcU8BcgEegEfAa0Lse6mwM7y9xO8y37L2bWmtILJy7wLeoIZJvZTDNbYWbP+LZgGgLZzrmicqxzrJklm1lyZmZmOeJKZffvVbuZuXwXdw1qzzmt6nsdRyRslffquOmUXujwOko/8acGOMdIYIZzrth3Owo4D3iI0vlA2gJj/Fmhc26icy7JOZcUFxcXyKwShvYcPMJvZ66mR8t63DW4vddxRMLaSbfVzawjpbuPRlF6wt9UwJxzg8q57l1AyzK3W/iWnchIfjyneRqQ4pzb6svyIdAXeAuoZ2ZRvq2OU61TBIAS38RMRSWO50f0IDryp8wmICLHnOpf0HpKty4ud84NdM69SOl1qsprKdDBNwoqhtJymHP8g8ysM1AfWHzcc+uZ2bFNhcHAOuecA74ArvUtvxGY7UcmqYLe+mYbi7bs4w9XdCGhUU2v44iEvVMVxzBgD/CFmb1uZkOAcp8l5dsiuIvSYyKpwDTn3Foze6LsJUwoLZQpvlI49txiSndTzTez1b6f+7rv7t8AD5jZZkqPebxZ3kxS9azbncPTn27g4i7xDE9qefoniMhpWZn/r0/8ALOalI6GGkXpJ//JwCzn3OfBjxcYSUlJLjk52esYUsHyC4u58qWFHMgr5LP7zqdBTZ3yI+IPM1vmnEs6fnl5RlUdds79yzl3BaXHFFZQ+qlfJKQ9/ekGNqbn8ux13VUaIgHk11FC59wB32ilIcEKJBIIX23M5K1vtjGmfwIXdNSoOpFA0vASqXQOHC7goekr6dC4Fo9c1tnrOCKVjk6dlUrFOcejM1dzIK+ASTf1IjZaEzOJBJq2OKRSmb4sjU/X7uWhiztxVrO6XscRqZRUHFJpbN93mD/NWUvftg249by2XscRqbRUHFIpFBWXcP/UFCIijOeG9yBSEzOJBI2OcUilMOGLLSzfkc0Lo3rSrF51r+OIVGra4pCwt2LHAV5YsImrejTjyu7NvI4jUumpOCSsHT5axP1TU2hSJ5YnrurqdRyRKkG7qiSs/fnf69i+P48pv+pLndhor+OIVAna4pCw9dnavUxZupPbL2hHn7YNvY4jUmWoOCQsZeTk88gHq+javA73X9jR6zgiVYqKQ8KOc46HZ6ziSGExz4/oSUyUfo1FKpL+xUnYmbx4O19uzOR3P0ukfeNaXscRqXJUHBJWNqUf4q8fpzKoUxy/7Nva6zgiVZKKQ8LG0aJi7p2SQq1qUTx9bXfMdHa4iBc0HFfCxnNzN7JuTw6v35BEXO1qXscRqbK0xSFhYfGWfUz8aiujerfioi7xXscRqdJUHBLyDuYV8uC0FBIa1uT3lyd6HUekytOuKgl5v5+9hvRDR5l5R39qxOhXVsRr2uKQkDY7ZRdzVu7mviEd6N6yntdxRAQVh4SwtAN5PPbhGpJa1+fOQe29jiMiPioOCUnFJY4Hpq3EORg/QhMziYQS7TCWkDTxq60s2bafZ6/rTssGNbyOIyJlaItDQs6aXQd5bu4Gft6tKdec09zrOCJyHBWHhJQjBcXcO2UFDWrG8L9Xd9XZ4SIhSLuqJKQ8+UkqWzIP8+4tfahXI8brOCJyAtrikJDxxfoMJi/ezi0D2zCwQyOv44jISag4JCRk5R7l4Rmr6NykNg9f0snrOCJyCtpVJZ5zzvHIB6vJOVLIu7f2JjY60utIInIK2uIQz01ZupN5qen8+tJOdG5Sx+s4InIaKg7x1NbMXJ74aB0D2zfi5gFtvI4jIuWg4hDPFBaXcP/UFGKiInj2uu5E6OxwkbCgYxzimRfnb2Jl2kFevv4cmtSN9TqOiJSTtjjEE8nf7+elLzZz7bkt+Fm3pl7HERE/qDikwh3KL+T+aSk0r1+dP1zRxes4IuKnoBaHmV1qZhvMbLOZPXKC+8ebWYrva6OZZZe5r7jMfXPKLB9iZst9yxeama63HWb+9NE6dh04wvjhPagdG+11HBHxU9COcZhZJDABuAhIA5aa2Rzn3Lpjj3HO3V/m8XcDPcus4ohzrscJVv0KMNQ5l2pmdwKPAWOC8BIkCD5evYcZy9K4Z3B7khIaeB1HRH6CYG5x9AY2O+e2OucKgCnA0FM8fhTwfjnW64Bjg/3rArvPKKVUmL0H83l05mq6t6jL3UM6eB1HRH6iYI6qag7sLHM7DehzogeaWWugDbCgzOJYM0sGioC/Oec+9C2/FfjYzI4AOUDfk6xzLDAWoFWrVmfwMiQQSkocD01fSUFRCeNH9CA6UofXRMJVqPzrHQnMcM4Vl1nW2jmXBPwCeN7M2vmW3w/8zDnXApgEPHeiFTrnJjrnkpxzSXFxccHMLuXw1jfbWLg5i8ev6ELbuFpexxGRMxDM4tgFtCxzu4Vv2YmM5LjdVM65Xb4/twL/AXqaWRzQ3Tn3ne9hU4H+AcwsQbB+bw5Pf7qBCxPjGdmr5emfICIhLZjFsRToYGZtzCyG0nKYc/yDzKwzUB9YXGZZfTOr5vu+ETAAWAccAOqaWUffQy8CUoP4GuQM5RcWc9+UFOpUj+apa7ppYiaRSiBoxzicc0VmdhfwGRAJvOWcW2tmTwDJzrljJTISmOKcc2Wengi8ZmYllJbb346NxjKzXwEf+O47AJKuTRIAAAj7SURBVNwcrNcgZ+7Zzzawfu8hJo3pRcNa1byOIyIBYD/+/7pySkpKcsnJyV7HqHIWbsril29+xw39WvPE0K5exxERP5nZMt+x5h8JlYPjUslk5xXw4PQU2sXV5NHLEr2OIyIBpIscSsA55/jtrNXsP1zAmzf2onqMJmYSqUy0xSEB98HyXXy8ei8PXNSJrs3reh1HRAJMxSEBtWNfHn+YvYbebRow9vy2XscRkSBQcUjAFBWX8MC0FCIijOeGdydSEzOJVEo6xiFn7MDhAr7YkMHslN0kbz/AP0b2oEX9Gl7HEpEgUXHIT7IlM5d569KZn5pB8vb9lDiIq12N+y7swNAezb2OJyJBpOKQcikqLiF5+4HSslifwbaswwAkNq3DXYPaMyQxnm7N62recJEqQMUhJ5WTX8iXGzKZl5rOfzZkcvBIITGREfRt15CbByQwODGe5vWqex1TRCqYikN+ZMe+POalpjMvNZ0l2/ZTVOJoUDOGCxPjuahLYwZ2iKNWNf3aiFRl+h+giisucaTsPMC81AzmrUtnU0YuAB0a1+LW89pyUZfG9GhZXyOkROQHKo4q6PDRIr7elMm81AwWrM9g/+ECoiKM3m0aMLJ3Ky5MbEzrhjW9jikiIUrFUUXszj7C/NR05qVmsHjLPgqKS6gTG8Wgzo25MDGe8zvGUbd6tNcxRSQMqDgqqZISx+pdB5mfms7c1AxS9+QAkNCwBjf0a82QxHiSEuprClcR8ZuKoxI5UlDMN5uzmL++9PyKjENHiTBIat2ARy/rzJDEeNrF1dRkSiJyRlQcYS4jJ5/56zOYn5rOws1Z5BeWUKtaFBd0jGNIYmMGdWpM/ZoxXscUkUpExRFmnHOs25PD/NTSsliZdhCA5vWqM7JXK4YkNqZPm4bERGkXlIgEh4ojDBwtKmbxln0/lMXug/mYQfcW9Xj4kk4MSWxMp/ja2gUlIhVCxRGi9uUeZcH6DOanZvD1pkwOFxRTPTqSgR0acd+FHRnUuTFxtTWHt4hUPBVHiHDOsTkjl7mppQe2l+84gHMQX6caQ3s256LEePq1a0hstGbTExFvqTg8VFhcwtJt+38oix378wDo2rwO9wzuwEVd4jmrWR3tghKRkKLiqGAH8wr5z8YM5q5L58uNmRzKLyImKoIB7Roy9vy2DElsTNO6unCgiIQuFUcF2JZ1uPREvHXpJG8/QHGJo1GtGC7r2oQhifGc16ERNWL0VyEi4UH/WwVBUXEJy3dk+87aTmdrZuncFZ2b1Ob2C9oyJDGeHi3qae4KEQlLKo4AOZRfyFcbs5ifms6CDRlk5xUSHWn0bduQG/qWXuKjZQNNpyoi4U/FcQZ27s/74cKB323bR2Gxo16NaAZ3asyQxHjO79iI2rG6cKCIVC4qDj+UlDhS0kp3Qc1bl8GG9EMAtI2ryc0D2jAkMZ5zWtUjShcOFJFKTMVxGnkFRXy9ybcLan0GWbkFREYYSa3r89jPExmSGE+bRpq7QkSqDhXHKfx21mpmLEujoKiE2rGlFw68qEs8F3SMo14NXThQRKomFccptKhfnev7tOKixHh6tWmguStERFBxnNKd/9Pe6wgiIiFHH6FFRMQvKg4REfGLikNERPyi4hAREb+oOERExC8qDhER8YuKQ0RE/KLiEBERv5hzzusMQWdmmcD2n/j0RkBWAONUdnq//KP3yz96v/xzpu9Xa+dc3PELq0RxnAkzS3bOJXmdI1zo/fKP3i//6P3yT7DeL+2qEhERv6g4RETELyqO05vodYAwo/fLP3q//KP3yz9Beb90jENERPyiLQ4REfGLikNERPyi4jgJM3vLzDLMbI3XWcKBmbU0sy/MbJ2ZrTWze73OFMrMLNbMlpjZSt/79SevM4UDM4s0sxVm9m+vs4Q6M/vezFabWYqZJQd03TrGcWJmdj6QC0x2znX1Ok+oM7OmQFPn3HIzqw0sA65yzq3zOFpIMjMDajrncs0sGlgI3Ouc+9bjaCHNzB4AkoA6zrnLvc4TyszseyDJORfwEya1xXESzrmvgP1e5wgXzrk9zrnlvu8PAalAc29ThS5XKtd3M9r3pU9xp2BmLYCfA294naWqU3FIwJlZAtAT+M7bJKHNt9slBcgA5jrn9H6d2vPAr4ESr4OECQd8bmbLzGxsIFes4pCAMrNawAfAfc65HK/zhDLnXLFzrgfQAuhtZtolehJmdjmQ4Zxb5nWWMDLQOXcOcBkwzrf7PSBUHBIwvn31HwDvOedmep0nXDjnsoEvgEu9zhLCBgBX+vbbTwEGm9m73kYKbc65Xb4/M4BZQO9ArVvFIQHhO9j7JpDqnHvO6zyhzszizKye7/vqwEXAem9ThS7n3KPOuRbOuQRgJLDAOfdLj2OFLDOr6RukgpnVBC4GAjZCVMVxEmb2PrAY6GRmaWZ2i9eZQtwAYDSlnwRTfF8/8zpUCGsKfGFmq4CllB7j0BBTCZR4YKGZrQSWAP/nnPs0UCvXcFwREfGLtjhERMQvKg4REfGLikNERPyi4hAREb+oOERExC8qDpEAMLPiMsOQU8zskQCuO0FXaZZQEuV1AJFK4ojv8iEilZ62OESCyDcnwtO+eRGWmFl73/IEM1tgZqvMbL6ZtfItjzezWb55OlaaWX/fqiLN7HXf3B2f+842F/GEikMkMKoft6tqRJn7DjrnugEvUXqFV4AXgXecc2cD7wEv+Ja/AHzpnOsOnAOs9S3vAExwzp0FZAPXBPn1iJyUzhwXCQAzy3XO1TrB8u+Bwc65rb6LQO51zjU0syxKJ74q9C3f45xrZGaZQAvn3NEy60ig9JIkHXy3fwNEO+f+EvxXJvLftMUhEnzuJN/742iZ74vR8UnxkIpDJPhGlPlzse/7RZRe5RXgeuBr3/fzgTvgh4me6lZUSJHy0qcWkcCo7pvN75hPnXPHhuTW910F9ygwyrfsbmCSmT0MZAI3+ZbfC0z0XY25mNIS2RP09CJ+0DEOkSDyHeNIcs5leZ1FJFC0q0pERPyiLQ4REfGLtjhERMQvKg4REfGLikNERPyi4hAREb+oOERExC//DyBsDYtqVkF+AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "tl_feature_extractor(epochs=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also run the fine-tuninig model for comparison:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "Train Loss: 0.8057; Accuracy: 0.7192\n",
      "Test Loss: 0.6171; Accuracy: 0.7790\n",
      "Epoch 2/5\n",
      "Train Loss: 0.5269; Accuracy: 0.8180\n",
      "Test Loss: 0.5153; Accuracy: 0.8221\n",
      "Epoch 3/5\n",
      "Train Loss: 0.4272; Accuracy: 0.8539\n",
      "Test Loss: 0.4016; Accuracy: 0.8589\n",
      "Epoch 4/5\n",
      "Train Loss: 0.3642; Accuracy: 0.8751\n",
      "Test Loss: 0.3933; Accuracy: 0.8642\n",
      "Epoch 5/5\n",
      "Train Loss: 0.3180; Accuracy: 0.8910\n",
      "Test Loss: 0.3768; Accuracy: 0.8712\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3de3hU9bn28e9DIIQECGACQjgFAUWCiEbQqruv2gPFVmytFhQVC1j3W6x2W7vdyt6XtWff3dba2oOAJwSptbXbWqzdpbbVVoEgchZBQAgHSQynBEhI8rx/zAqOcYAJZLLmcH+ua67MrLVm8jiSuef3W4fH3B0REZHm2oVdgIiIJCcFhIiIxKSAEBGRmBQQIiISkwJCRERiah92Aa2loKDABw4cGHYZIiIpZenSpZXuXhhrXdoExMCBAykrKwu7DBGRlGJm7xxtnaaYREQkJgWEiIjEpIAQEZGYFBAiIhKTAkJERGJSQIiISEwKCBERiSltzoMQEckk+w8dZvX2fazatpfe+Z24/Kzerf47FBAiIklu74HDrNq+l1Xb9rJy215Wb9/HpsqaI+s/M7KPAkJEJN1V1dRFBUHk59aqg0fWF3XrRElRVz43qoiSvvmU9MmnsEvHhNSigBARCUnF/lpWbfvgyGDbnvfDoH+PXEYU5TNxdH9K+uRTUpRPj7zsNqtPASEikmDuzrv7aj80Mnh3X+2RbYoL8jhnQHduuGAAI4ryGd4nn/zcDiFWrYAQEWlV7s72vYdYWf5+EKzato/K6kgYmMFphZ25YNAplBRFRgXD+3SlS064YRCLAkJE5AS5O1urDrLqSBBEpomqauoAaGcwpGcXPjq0kJKirowoymdY767kdUyNj97UqFJEJGSNjc47VQciU0Tb3g+EfYfqAWjfzhjaqwsfG9YzMkVUlM+wU7vSKTsr5MpPnAJCRKSZhkZnU2U1q7btOxIEa7bvY39tJAyys9pxRu8uXH5WnyMjg6G9upDTIXXDIBYFhIhktPqGRjZURMKg6YiiNTv2caCuAYCO7dsxrHdXrhxVRElRV0qK8hnSswvZ7dP/QhQKCBHJGHX1jazftT8IgsjoYO2OfdTWNwLQqUMWw/t05ZrSfsEO5K4MLuxM+6z0D4NYFBAikpZq6xtYt3P/kSBYvX0vb+7YT11DJAw6d2zPmX26Mun8AUemiYoLOpPVzkKuPHkoIEQk5R063MDaHfs+MDJ469391Dc6AF1z2lNSlM/kCwdGRgZ9ujLwlDzaKQyOSQEhIinlQF09a4KL1K3cto/V2/eyflc1DUEYdMvtwIiifKadPoiSPvmMKMqnX49OmCkMWkoBISJJa/+hw6zZvu/IZShWbtvL2xXVeCQLKOicTUlRPh8b1uvIPoOibgqD1qKAEJHQ1dY3sOW9A2ysrGFjRQ1rduxj9ba9bIy6Ymmvrh0p6ZPP5SN6U1IUGRn06tpRYZBACggRaRONjc6OfYfYWFHNpiAINlXWsLGymm27DxLMEAHQJz+HkqJ8rhxVFJx01pWeXXLCKz5DKSBEpFXtrqljY2Xw4R+EQdOt6XBSgNzsLIoL8ji7X3c+O6ovgwryKC7IY2BBHvmdku+6RJlIASEiLXbocMMHPvg3VkRGApsqa9hz4PCR7dq3M/r3yKW4II+LBhdQXJjHoILODCrMo2cXTQ8lOwWEiMTU0Ohs233wyAd/05TQpsqaD/QsgMj+geKCPMaN6H1kJFBckEe/Hrl0yNCTzNKBAkIkg7k779XUBR/+1ZGpoSAI3nnvwJGTygC6dGzPoMI8zhvYnWsK+gWjgciUUOcUuTqptIz+r4pkgJra+mZTQsGooLKG/cHVSAE6ZBkDTol8+790WM9gNNCZ4oI8Cjpna0oowyQ0IMxsLPBjIAuY5e7fa7a+P/A40C3Y5i53XxCsOwv4JdAVaATOc/dDiaxXJJUdbmikfPfBD3z4bwr2DUR3LoNIX+PigjyuPLsoMh1UmMdpBZ3p0y0nY687JB+WsIAwsyzgIeDjQDmwxMyec/c1UZvNAJ5295+b2ZnAAmCgmbUHngSud/flZnYKcBiRDOfu7Npf+/4holFHCW2pOnDk0hIA+Z06MKgwjwsHF3BaYecj+wUGnpKX0j0KpO0kcgQxGtjg7hsBzGw+MB6IDggnMkIAyAe2B/c/Aaxw9+UA7v5eAusUSTr7Dh0+si9g45GpoWo2VdRQE1yGGiC7fTuKT8nj9FO7MLbkVIoL8hgUHCnUvQ2b20t6SmRAFAFbox6XA2OabXMv8CczuxXIAz4WLB8KuJm9CBQC8939/ua/wMxuBm4G6N+/f6sWL5JotfUNbK06EDUaeD8QmvoXQ6SHcd/unSgu6EzpgB4MKnz/KKE++Z10wTlJmLB3Uk8EHnP3H5jZBcAcMysJ6roIOA84ACw0s6XuvjD6ye7+MPAwQGlpqSOShKpq6lizfd/7RwkFt61VBz5w9vApedkMKszj0jMKj+wYPq0wcqhounUqk9SQyIDYBvSLetw3WBZtCjAWwN1fNbMcoIDIaOPv7l4JYGYLgHOAhYikkFfffo+bHlvMocPvN6QpLsijpCifK0b2CUYDnSk+JY/8XJ09LMklkQGxBBhiZsVEgmECcG2zbbYAlwGPmdkwIAeoAF4Evm5muUAd8FHgRwmsVaTVLX2niimPL6Ff91zuvWI4gwrzOLVrjg4VlZSRsIBw93ozm07kwz4LeMTdV5vZfUCZuz8H3AHMNLOvEtlhPdndHdhtZj8kEjIOLHD3PySqVpHWtqJ8D5MfWULPLh2ZO3UMPbvqQnOSesw9PabuS0tLvaysLOwyRFi7Yx8THn6NLjntefpLF9CnW6ewSxI5qmD/bmmsdTojRqQVbdi1n0mzFtGpQxZPTTtf4SApTQEh0ko2V9Zw7cxFmBnzpo2hX4/csEsSOSkKCJFWUL77ANfNWsThhkbmTh3DoMLOYZckctIUECInaefeQ1w7cxH7Dx1mzpQxnH5ql7BLEmkVYZ8oJ5LSKqtruW7Wa1TV1DFnymhKivLDLkmk1WgEIXKCdtfUMWnWIrbtOcgjk89jVP/uYZck0qo0ghA5AXsPHuaGRxazsbKGR248j9HFPcIuSaTVaQQh0kLVtfXc9Ohi3ty5j19MOoeLhhSEXZJIQmgEIdICB+samPLYEpaX7+Wha0dx6Rm9wi5JJGE0ghCJ06HDDdw8p4zFm6v44TUjGVvSO+ySRBJKASESh7r6RqbPe52X11fy/avOYvzZRWGXJJJwCgiR46hvaOT2Xy3jz2t38c3xw7mmtN/xnySSBhQQIsfQ2Ojc+cwKFqzcyYzLh3H9BQPDLkmkzSggRI7C3bnndyt5dtk2vvaJoUy9eFDYJYm0KQWESAzuzjd+v4anFm9l+iWDmX7pkLBLEmlzCgiRZtyd7/3xTR7752amXFTMHZ8YGnZJIqFQQIg088Cf1/PLv21k0vn9mXH5MLUIlYylgBCJ8vO/vs2PF67n6nP7ct8VJQoHyWgKCJHAI69s4vt/fJMrRvbhe1edRbt2CgfJbAoIEWDeoi3c9/waPjm8Fz+4ZiRZCgcRBYTIb5aWc8/vVnLJ6YX8ZOI5dMjSn4UIKCAkwz2/Yjt3PrOcj5x2Cj+fdC7Z7fUnIdJEfw2Ssf60eie3z3+D0gE9mHlDKTkdssIuSSSpKCAkI/3trQqmz1vG8KJ8Zk8uJTdbV74XaU4BIRnnn29XcvMTZQzu2ZknbhpNl5wOYZckkpQUEJJRyjZXMfXxMvr3yGXOlNHk5yocRI5GASEZY0X5Hm56dAm9uuYwd9oYTuncMeySRJKaAkIywprt+7h+9mLyczswd+oYenbJCbskkaSngJC0t/7d/Vw/exG52Vk8Ne18+nTrFHZJIilBASFpbXNlDdfNWkS7dsbcqWPo1yM37JJEUoYCQtJW+e4DXDvzNeobnblTxzCosHPYJYmkFAWEpKWdew9x7cxFVNfWM2fKaIb26hJ2SSIpR2cHSdqp2F/LtbNeo6qmjienjmF4n/ywSxJJSRpBSFrZXVPH9bMXsWPPIR6ZfB5n9+sWdkkiKSuhAWFmY81snZltMLO7Yqzvb2YvmdkyM1thZuNirK82s68lsk5JD3sPHub6RxaxsbKGWTeWMrq4R9gliaS0hAWEmWUBDwGfAs4EJprZmc02mwE87e6jgAnAz5qt/yHwQqJqlPRRXVvP5EcXs27nfn456VwuHFwQdkkiKS+RI4jRwAZ33+judcB8YHyzbRzoGtzPB7Y3rTCzK4FNwOoE1ihp4GBdA198bAkryvfyk4nncMkZPcMuSSQtJDIgioCtUY/Lg2XR7gUmmVk5sAC4FcDMOgP/DnzjWL/AzG42szIzK6uoqGituiWFHDrcwM1zyliyuYofXjOSsSWnhl2SSNoIeyf1ROAxd+8LjAPmmFk7IsHxI3evPtaT3f1hdy9199LCwsLEVytJpa6+kS/PfZ2X11dy/1VnMf7s5t8/RORkJPIw121Av6jHfYNl0aYAYwHc/VUzywEKgDHA583sfqAb0Ghmh9z9pwmsV1JIfUMjt/9qGQvf3MU3ryzh6tJ+x3+SiLRIIgNiCTDEzIqJBMME4Npm22wBLgMeM7NhQA5Q4e4XN21gZvcC1QoHadLQ6Nz5zAoWrNzJjMuHcf35A8IuSSQtJWyKyd3rgenAi8BaIkcrrTaz+8zsimCzO4BpZrYceAqY7O6eqJok9TU2Ovc8u5Jnl23jzk+eztSLB4VdkkjasnT5PC4tLfWysrKwy5AEcnfufW41j7/6DtMvGczXPnl62CWJpDwzW+rupbHWhb2TWiQu7s73XniTx199h6kXFXPHJ4aGXZJI2lNASEr40Z/X88u/b+T68wdwz+XDMLOwSxJJewoISXo/++sGHly4nmtK+/KNK4YrHETaiAJCktrsVzZx/x/XMf7sPnz3c2fRrp3CQaStKCAkac1d9A7ffH4NY4efyg+uHkmWwkGkTSkgJCk9s7Sce55dxaVn9OTBiaNon6V/qiJt7bh/dWZ2q5l1b4tiRAB+v3w7X39mORcNLuBn151DdnuFg0gY4vnL6wUsMbOng/4OGudLwvxp9U5u/9UblA7owcM3nEtOh6ywSxLJWMcNCHefAQwBZgOTgfVm9h0zOy3BtUmG+eu6XUyft4wRRfnMnlxKbrY64oqEKa6xe3D5i53BrR7oDjwTXExP5KT9c0MlX5qzlME9O/P4TaPpktMh7JJEMt5xv6KZ2W3ADUAlMAu4090PB5flXg98PbElSror21zFlMfLGHBKLk9OHUN+rsJBJBnEM4bvAXzO3d+JXujujWb26cSUJZli+dY9TH50Cb3zc3hy6hh65GWHXZKIBOKZYnoBqGp6YGZdzWwMgLuvTVRhkv7WbN/HDY8spnteB+ZOG0PPLjlhlyQiUeIJiJ8D0Z3dqoNlIids/bv7mTR7EbnZWcybej698zuFXZKINBNPQFh0jwZ3bySxjYYkzW2qrOHaWYvIamfMm3Y+/Xrkhl2SiMQQT0BsNLOvmFmH4HYbsDHRhUl62lp1gOtmvkZDozNv6hiKC/LCLklEjiKegLgF+AiRtqHlRPpF35zIoiQ97dx7iOtmLaK6tp45U0YzpFeXsEsSkWM47lSRu+8i0k9a5IRV7K/l2lmvUVVTx5NTxzC8T37YJYnIccRzHkQOMAUYDhw5zMTdv5jAuiSNVNXUMWnWInbsOcQTU0Zzdr9uYZckInGIZ4ppDnAq8Engb0BfYH8ii5L0sffgYa6fvYhN79Uw68ZSzhvYI+ySRCRO8QTEYHf/T6DG3R8HLieyH0LkmKpr65n86GLeenc/v7z+XC4cXBB2SSLSAvEExOHg5x4zKwHygZ6JK0nSwcG6Br742BJWlO/lJxPP4ZLT9U9GJNXEcz7Dw0E/iBnAc0Bn4D8TWpWktEOHG5j2RBllm6t4YMIoxpacGnZJInICjhkQwQX59rn7buDvwKA2qUpSVl19I/937uu8sqGS/756JFeM7BN2SSJygo45xRScNa2rtUpc6hsauW3+Mv7y5i6+dWUJnz+3b9glichJiGcfxJ/N7Gtm1s/MejTdEl6ZpJSGRudrv17OC6t2MuPyYUw6f0DYJYnISYpnH8QXgp9fjlrmaLpJAo2Nzt2/Xcnv3tjOnZ88nakX65+GSDqI50zq4rYoRFKTu3Pv71fzq7Kt3HrpYL58yeCwSxKRVhLPmdQ3xFru7k+0fjmSStyd777wJk+8+g7TLi7m3z4+NOySRKQVxTPFdF7U/RzgMuB1QAGR4X70v2/x8N83csMFA7h73DDMLOySRKQVxTPFdGv0YzPrBsxPWEWSEh56aQMP/mUDXyjtx72fGa5wEElD8RzF1FwNoP0SGWz2K5v4fy+uY/zZffjO50bQrp3CQSQdxbMP4vdEjlqCSKCcCTydyKIkeT2/YjvffH4Nnyo5lR9cPZIshYNI2opnH8R/R92vB95x9/IE1SNJbNW2vXzt18s5d0B3HphwNu2zTmQAKiKpIp6/8C3AInf/m7v/A3jPzAbG8+JmNtbM1pnZBjO7K8b6/mb2kpktM7MVZjYuWP5xM1tqZiuDn5e24L9JEmDX/kNMe6KMHrnZ/GLSuXRsnxV2SSKSYPEExK+BxqjHDcGyYzKzLOAh4FNEpqUmmtmZzTabATzt7qOIdK37WbC8EviMu48AbiTSk0JCUlvfwC1zlrLnwGFm3lhKYZeOYZckIm0gnoBo7+51TQ+C+9lxPG80sMHdNwbPmQ+Mb7aNA12D+/nA9uB3LHP37cHy1UAnM9OnUgjcnbt/u4rXt+zhB9eMVKtQkQwST0BUmNkVTQ/MbDyRb/jHUwRsjXpcHiyLdi8wyczKgQXArXzYVcDr7l7bfIWZ3WxmZWZWVlFREUdJ0lKzXt7Eb14v57bLhjBuRO+wyxGRNhRPQNwC3G1mW8xsC/DvwJda6fdPBB5z977AOGBOcIlxAMxsOPD9o/0+d3/Y3UvdvbSwsLCVSpImL63bxXdfWMunSk7ltsuGhF2OiLSxeE6Uexs438w6B4+r43ztbUC/qMd9g2XRpgBjg9d91cxygAJgl5n1BZ4FbghqkDa0YVc1X5m3jDNO7coPrhmpcx1EMtBxRxBm9h0z6+bu1e5ebWbdzexbcbz2EmCImRWbWTaRndDPNdtmC5FLd2Bmw4hcyqMiOFv7D8BdwZFT0ob2HjjMtCfK6NihHTNvLCU3O56joUUk3cQzxfQpd9/T9CDoLjfueE9y93pgOvAisJbI0Uqrzey+qH0adwDTzGw58BQw2d09eN5g4L/M7I3gpqbGbaC+oZEvz3ud8t0H+MWkcynq1inskkQkJPF8Ncwys45NO4nNrBMQ1xFF7r6AyM7n6GX/FXV/DXBhjOd9C4hnlCKt7Ft/WMsrGyq5/6qzKB2ovlAimSyegJgLLDSzRwEDJgOPJ7IoCcf8xVt47J+b+eKFxVxzXr/jP0FE0lo8O6m/H0wBfYzIeQsvAuonmWYWb6riP/9nFf8ytJC7x50RdjkikgTivZjOu0TC4WrgUiL7FCRNlO8+wL8+uZR+3XP5ycRRusaSiADHGEGY2VAi5ylMJHJi3K8Ac/dL2qg2aQM1tfVMfbyMuoZGZt5YSn6nDmGXJCJJ4lhTTG8CLwOfdvcNAGb21TapStpEY6Pzb0+/wVvv7ufRm0ZzWmHnsEsSkSRyrLmEzwE7gJfMbKaZXUZkJ7WkiQf+/BYvrn6Xu8cN46NDdSa6iHzQUQPC3X/n7hOAM4CXgNuBnmb2czP7RFsVKInx/IrtPPiXDVx9bl+mXKQGgSLyYcfdG+nuNe4+z90/Q+RyGcuIXI9JUlR0459vfbZE/aRFJKYWHa7i7ruDC+RdlqiCJLHU+EdE4qWL7GSQ6MY/v77lAjX+EZFjUkBkiOjGPz+77hxKitT4R0SOTWdEZYjZr6jxj4i0jAIiA7y0bhffWaDGPyLSMgqINNfU+Od0Nf4RkRZSQKSxpsY/2e3bMfOGc9X4R0RaRJ8YaSq68c9T086nb/fcsEsSkRSjgEhT316gxj8icnI0xZSGfrVkC4/+Q41/ROTkKCDSzJLNVcz43SouHlKgxj8iclIUEGmkfPcBbpkTafzz04nnqPGPiJwUfYKkiQ81/slV4x8ROTkKiDQQ3fjnp9eeo8Y/ItIqFBBp4IGF69X4R0RanQIixf1hxQ4eXLhejX9EpNUpIFLYqm17uePXb6jxj4gkhAIiRanxj4gkms6kTkFq/CMibUEBkWLcnXueVeMfEUk8TTGlmNmvbOKZpWr8IyKJp4BIIWr8IyJtSQGRItT4R0TamgIiBajxj4iEQZ80Sa6+oZHpT6nxj4i0PQVEkvv2grW8vF6Nf0Sk7SV0isnMxprZOjPbYGZ3xVjf38xeMrNlZrbCzMZFrfuP4HnrzOyTiawzWanxj4iEKWEjCDPLAh4CPg6UA0vM7Dl3XxO12QzgaXf/uZmdCSwABgb3JwDDgT7An81sqLs3JKreZKPGPyIStkSOIEYDG9x9o7vXAfOB8c22caBrcD8f2B7cHw/Md/dad98EbAheLyOo8Y+IJINEfvIUAVujHpcHy6LdC0wys3Iio4dbW/BczOxmMyszs7KKiorWqjtUNbX1THtiqRr/iEjowv5qOhF4zN37AuOAOWYWd03u/rC7l7p7aWFh6vdBaGx07nh6Oet27lPjHxEJXSKPYtoGRO9Z7RssizYFGAvg7q+aWQ5QEOdz084DC9fzx9U7mXG5Gv+ISPgSOYJYAgwxs2Izyyay0/m5ZttsAS4DMLNhQA5QEWw3wcw6mlkxMARYnMBaQ9fU+OfzavwjIkkiYSMId683s+nAi0AW8Ii7rzaz+4Ayd38OuAOYaWZfJbLDerK7O7DazJ4G1gD1wJfT+Qim6MY/31bjHxFJEhb5PE59paWlXlZWFnYZLVaxv5bxP30FgP+ZfpF6O4hImzKzpe5eGmudzqQOUW19A1+aU8ZuNf4RkSSkgAiJGv+ISLIL+zDXjKXGPyKS7BQQIfirGv+ISApQQLSxtyuqufUpNf4RkeSngGhDew8cZurjZWRnqfGPiCQ/fUK1ETX+EZFUo4BoI2r8IyKpRlNMbaCp8c9NFw5U4x8RSRkKiASLbvxzz7hhYZcjIhI3BUQCqfGPiKQyfWIliBr/iEiqU0AkgBr/iEg6UEAkwI+Dxj93j1PjHxFJXQqIVvaHFTv4sRr/iEgaUEC0IjX+EZF0ooBoJRX7a7n5iTJ65Gbzi0nn0rF9VtgliYicFJ1J3QqaGv9UHajjmVs+osY/IpIWFBAnKbrxz0PXqvGPiKQPTTGdpKbGP1+5bAiXn6XGPyKSPhQQJ+Fvb1Ucafxzuxr/iEiaUUCcoLcrqpk+73U1/hGRtKWAOAFq/CMimUCfbC0U3fhnnhr/iEgaU0C00HcWvMnL6yv5/lUjOE+Nf0QkjWmKqQWeXrKVR/6xiZsuHMgXzusfdjkiIgmlgIjTks1V3PO7lWr8IyIZQwERBzX+EZFMpE+64zhQp8Y/IpKZFBDHEN345ycTR6nxj4hkFAXEMfx44XpeWBVp/PN/Tu8ZdjkiIm1KAXEUavwjIplOARGDGv+IiCggPqSp8U93Nf4RkQyX0IAws7Fmts7MNpjZXTHW/8jM3ghub5nZnqh195vZajNba2YPWht8ja+tb+CWJ5dSdaCOmTeUqvGPiGS0hF1qw8yygIeAjwPlwBIze87d1zRt4+5fjdr+VmBUcP8jwIXAWcHqV4CPAn9NVL3uzoxnV7H0nd1q/CMiQmJHEKOBDe6+0d3rgPnA+GNsPxF4KrjvQA6QDXQEOgDvJrBWZr+yiV+r8Y+IyBGJDIgiYGvU4/Jg2YeY2QCgGPgLgLu/CrwE7AhuL7r72hjPu9nMysysrKKi4oQLVeMfEZEPS5ad1BOAZ9y9AcDMBgPDgL5EQuVSM7u4+ZPc/WF3L3X30sLCwhP6xWr8IyISWyIDYhvQL+px32BZLBN4f3oJ4LPAa+5e7e7VwAvABYkoMqdDFmf366bGPyIizSQyIJYAQ8ys2MyyiYTAc803MrMzgO7Aq1GLtwAfNbP2ZtaByA7qD00xtYaibp2YM2WMGv+IiDSTsIBw93pgOvAikQ/3p919tZndZ2ZXRG06AZjv7h617BngbWAlsBxY7u6/T1StIiLyYfbBz+XUVVpa6mVlZWGXISKSUsxsqbuXxlqXLDupRUQkySggREQkJgWEiIjEpIAQEZGYFBAiIhKTAkJERGJKm8NczawCeOckXqIAqGylcjKB3q+W0fvVMnq/WuZk3q8B7h7zWkVpExAny8zKjnYssHyY3q+W0fvVMnq/WiZR75emmEREJCYFhIiIxKSAeN/DYReQYvR+tYzer5bR+9UyCXm/tA9CRERi0ghCRERiUkCIiEhMGR0QZvaIme0ys1Vh15IKzKyfmb1kZmvMbLWZ3RZ2TcnMzHLMbLGZLQ/er2+EXVMqMLMsM1tmZs+HXUsqMLPNZrbSzN4ws1bteZDR+yDM7F+AauAJdy8Ju55kZ2a9gd7u/rqZdQGWAle6+5qQS0tKZmZAnrtXB50RXwFuc/fXQi4tqZnZvwGlQFd3/3TY9SQ7M9sMlLp7q59YmNEjCHf/O1AVdh2pwt13uPvrwf39RDoFFoVbVfLyiOrgYYfglrnfyOJgZn2By4FZYdciGR4QcuLMbCAwClgUbiXJLZgueQPYBfyvu+v9OrYHgK8DjWEXkkIc+JOZLTWzm1vzhRUQ0mJm1hn4DXC7u+8Lu55k5u4N7n420BcYbWaayjwKM/s0sMvdl4ZdS4q5yN3PAT4FfDmYOm8VCghpkWAu/TfAXHf/bdj1pAp33wO8BIwNu5YkdiFwRTCnPh+41MyeDLek5Ofu24Kfu4BngdGt9doKCIlbsNN1NrDW3X8Ydj3JzswKzaxbcL8T8HHgzXCrSl7u/h/u3tfdBwITgL+4+6SQy0pqZpYXHDCCmeUBnwBa7ajMjA4IM3sKeBU43czKzWxK2DUluQuB64l8s3sjuI0Lu6gk1ht4ycxWAEuI7IPQoZvSmnoBr5jZcmAx8Ad3/2NrvXhGH+YqIiJHl9EjCBEROXLzcLEAAAGaSURBVDoFhIiIxKSAEBGRmBQQIiISkwJCRERiUkCItICZNUQd4vuGmd3Viq89UFcWlmTSPuwCRFLMweDSGSJpTyMIkVYQXJP//uC6/IvNbHCwfKCZ/cXMVpjZQjPrHyzvZWbPBr0ilpvZR4KXyjKzmUH/iD8FZ2CLhEIBIdIynZpNMX0hat1edx8B/JTIVUkBfgI87u5nAXOBB4PlDwJ/c/eRwDnA6mD5EOAhdx8O7AGuSvB/j8hR6UxqkRYws2p37xxj+WbgUnffGFzQcKe7n2JmlUSaLB0Olu9w9wIzqwD6untt1GsMJHI5jiHB438HOrj7txL/XybyYRpBiLQeP8r9lqiNut+A9hNKiBQQIq3nC1E/Xw3u/5PIlUkBrgNeDu4vBP4VjjQVym+rIkXipW8nIi3TKegQ1+SP7t50qGv34MqttcDEYNmtwKNmdidQAdwULL8NeDi4gnADkbDYkfDqRVpA+yBEWkEiG8eLhEVTTCIiEpNGECIiEpNGECIiEpMCQkREYlJAiIhITAoIERGJSQEhIiIx/X+/Z/4H4MYKbgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "tl_fine_tuning(epochs=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The feature extracting model achieves high validation accuracy even after the first epoch, but taps out after that. That's because the model capacity is limited, since only the final fully-connected layer weights are trainable. On the other hand the fine-tuning model uses it's full capacity and achieves higher overall accuracy. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
